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SUMMARY 


Problem 


In  order  to  ensure  Fleet  readiness,  the  U.S.  Navy  is  continually  seeking 
new  approaches  for  assessing  the  job  performance  of  its  personnel.  While 
many  partial  performance  measurement  systems  exist,  the  Navy  does  not  now 
have  a comprehensive  system  for  the  measurement  of  job  performance  profici- 
ency. Several  shortcomings  in  performance  measurement  efforts  have  been 
noted  (Pickering  & Anderson,  1977).  A comprehensive  and  cost-effective 
system  is  needed  to  generate  performance  measurement  data  that  are  readily 
understood  by  Navy  personnel  managers.  NAVPERSRANDCEN  has  proposed  a Per- 
formance Proficiency  Assessment  System  (PPAS)  in  response  to  this  need. 

Objectives 

Previous  studies  have  identified  numerous  performance  measures  for  ASW  team 
training.  The  objectives  of  this  study  were  to  (1)  identify  Surface  Sonar  Tech- 
nician (STG)  performance  measures  obtainable  from  the  14A2  ASW  Team  Trainer 
Complex  that  might  provide  data  useful  to  Navy  personnel  managers;  (2)  identify 
procedures  for  obtaining  the  desired  data  during  team  trainer  exercises;  (3) 
develop  appropriate  procedures  for  analyzing,  summarizing,  and  presenting  the 
data;  and  (4)  evaluate  methods  of  implementing  an  automated  data  collection 
system  for  the  14A2  Complex  that  could  be  used  in  a more  comprehensive  PPAS. 

Approach 


A review  of  ASW-related  documentation  was  conducted,  and  questionnaires 
were  used  to  obtain  information  from  ASW  instructors.  Candidate  performance 
measures  were  selected  for  analysis  and  evaluation.  Computer  data  collection 
programs  were  developed  to  collect  the  data  from  the  14A2E  ASW  Team  Trainer  and 
the  associated  14E19  Sonar  Trainer.  Data  were  collected  from  six  ASW  teams 
on  three  separate  exercises  per  team.  The  data  from  these  18  exercises  were 
analyzed,  and  each  performance  measure  was  evaluated  to  determine  its  applicability 
in  a Performance  Proficiency  Assessment  System. 

Conclusions 


1.  Data  from  the  14A2  Trainer  Complex  can  provide  inputs  to  a Performance 
Proficiency  Assessment  System. 


2.  Procedures  can  be  developed  for  summarizing  and  presenting  proficiency/ 
deficiency  data  so  that  they  can  be  readily  understood  by  Navy  personnel 
managers. 

3.  Procedures  can  be  developed  for  automating  the  collection  and  analysis 
of  the  desired  information. 

Future  Actions 


1.  Experiments  will  be  carried  out  to  determine  the  degree  to  which  the 
types  of  performance  information  described  in  this  report  would  be  useful  as 
part  of  a Performance  Proficiency  Assessment  System. 
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2.  If  it  is  determined  that  such  information  is  useful,  an  effort  will 
be  made  to  identify  the  steps  that  must  be  taken  to  automate  the  collection 
and  analysis  of  performance  data  from  the  14A2  Trainer  Complex.  Such  auto- 
mation would  be  based  upon  the  procedures  and  techniques  outlined  in  this 
report. 
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INTRODUCTION 


Probl em 

Fleet  readiness  is  directly  dependent  on  the  Navy's  ability  to  provide 
trained  personnel  who  are  capable  of  operating  its  ships,  aircraft,  and 
weapons  systems.  For  this  reason,  the  Navy  must  continually  seek  new  approaches 
to  assessing  the  degree  to  which  Fleet  personnel  are  trained  to  perform  the 
critical  aspects  of  their  jobs.  While  many  partial  performance  measurement 
systems  exist,  the  Navy  does  not  now  have  a comprehensive  system  for  measuring 
job  performance  proficiency.  Current  procedures  fall  short  for  the  following 
reasons : 


1.  Evaluators  often  are  not  trained  in  performance  testing  methods. 

2.  Personnel  are  usually  evaluated  by  subjective  instructor  ratings  or 
paper-and-pencil  tests. 

3.  Evaluations  tend  to  be  general  and  do  not  identify  specific  deficiencies. 

4.  Operational  requirements  take  precedence  over  performance  measurement. 

5.  Frequently,  evaluations  are  compromised  for  practical  or  administrative 
reasons. 

A comprehensive,  cost-effective  system  is  needed  that  will  provide  job 
performance  measurement  data  that  can  be  readily  understood  by  Navy  personnel 
managers. 

Background 

The  Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCEN)  is  in- 
vestigating the  feasibility  of  developing  a Performance  Proficiency  Assessment 
System  (PPAS) . In  1976,  a NAVPERSRANDCEN  study  proposed  that  a prototype  PPAS 
should  be  patterned  after  industrial  quality  control  methods.  In  this  approach, 
relatively  small  samples  of  a product  are  tested  periodically;  and  when  de- 
ficiencies are  found,  appropriate  corrective  actions  are  taken  (Pickering  & 
Anderson,  1976).  A clear  distinction  was  made  between  the  concept  of  a PPAS 
and  other  performance  measurement  processes  under  consideration:  "Such  a system 
would  not  be  concerned  with  evaluating  individuals  or  Navy  units.  It’s  purpose 
would  be  to  supply  appropriate  personnel  managers  with  information  on  how  well 
their  systems  are  working."  Using  the  guidelines  delineated  in  this  study, 
NAVPERSRANDCEN  began  an  advanced  development  effort  aimed  at  producing  a proto- 
type PPAS  for  the  STG  (Surface  Sonar  Technician)  rating. 

In  1977,  Honeywell's  Training  and  Control  Systems  Center  was  funded  by 
NAVPERSRANDCEN  to  develop  performance  measurement  procedures  for  the  14A2  ASW 
Team  Trainer.1  That  study  was  in  support  of  another  Center  effort  that  is  con- 
cerned with  developing  improved  techniques  for  providing  ASW  teams  with  immediate 

^he  14A2  Team  Trainer  is  used  for  refresher  training  of  Fleet  ASW  teams. 
Trainers  are  located  at  Pearl  Harbor,  Hawaii;  San  Diego,  California;  Charleston, 
South  Carolina;  and  Norfolk,  Virginia. 
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feedback  on  their  performance. 2 In  1978,  Honeywell  was  funded  to  expand  the 
scope  of  their  effort  to  consider  how  the  trainers  could  be  utilized  in  support 
of  a PPAS  for  the  STG  rating.  Specifically,  the  expanded  effort  evaluated  per- 
formance measures  related  to  the  sonar  and  underwater  battery  (UB)  plot  subteams. 
This  report  will  describe  those  aspects  of  the  Honeywell  efforts  that  are 
pertinent  to  the  development  of  a prototype  PPAS. 

Objectives 

1.  Identify  STG  performance  measures  that  should  provide  useful  diagnostic 
deficiency  data. 

2.  Identify  STG  measurement  data  available  from  the  14A2  Complex  and  the 
requirements  for  collecting  them. 

3.  Collect  data  during  selected  ASW  exercises  and  use  them  to  evaluate 
the  applicability  of  potential  performance  measures  in  the  PPAS. 

4.  Assess  the  feasibility  of  collecting  these  data  operationally. 

5.  Define  appropriate  procedures  for  preparing  and  presenting  the  data  for 
use  by  Navy  personnel  managers. 

6.  Evaluate  methods  of  implementing  an  automated  data  collection  system 
for  the  14A2  Complex  that  could  be  used  with  a more  comprehensive  PPAS. 


2For  a full  description  of  that  study,  see  the  final  report  on  it  (Bell, 
1979). 
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APPROACH 


* 


- 


Data  Collection 


First,  questionnaires  were  administered  to  14A2  instructors  at  the  Fleet 
Anti-Submarine  Warfare  Training  Center,  Pacific  (FLEASWTRACENPAC) , San  Diego. 

The  questionnaires  were  designed  to  assess  the  relevance  and  importance  of 
potential  performance  measures  and  to  determine  the  most  appropriate  techniques 
for  collecting  data. 

Next,  two  automated  data  collection  programs  and  a data  reduction  program 
were  developed.  Data  were  collected  during  ASW  exercises  using  the  14A2E 
Sonar  Trainer3  in  independent  mode  and  in  joint  mode  with  the  14E19;  in 
independent  mode  the  AN/SQS-23  Sonar  was  simulated  and  in  joint  mode  the 
AN/SQS-26CX  Sonar  was  simulated.  Figure  1 provides  a flow  diagram  of  the  data 
collection  process. 

A 14A2  Data  Collection  Program  was  developed  as  a subroutine  of  the  14A2 
Main  Trainer  Program;  it  collected  data  from  the  XDS-930  Computer  on  the  14A2 
Magnetic  Tape  Unit  (MTU)  for  all  simulated  vehicle  motion,  SQS-23  tracking  error, 
and  target  motion  analysis  (TMA) . It  also  recorded  problem  event  data,  including 
the  time  of  events  such  as  weapon  assignment,  weapon  fire,  and  instructor  con- 
trol settings. 

A 14E19  Data  Collection  Program  was  developed  to  provide  data  on  the  opera- 
tion of  the  14E19  Trainer  when  it  was  run  in  joint  mode  with  the  14A2E.  This 
program  provided  sonar  tracking  data  as  well  as  switch  or  control  data  for  the 
SQS-26CX  Sonar  when  it  was  used  in  place  of  the  SQS-23.  Since  it  was  not  pos- 
sible to  collect  data  directly  in  the  14E19  Trainer,  a program  was  designed  to 
process  data  from  the  14E19  through  the  Universal  Interface  Unit  for  recording 
on  the  computer  system  associated  with  the  14E23  Trainer,  another  trainer 
available  at  FLEASWTRACENPAC.  (A  few  exercises  not  originally  planned  for  were 
recorded  using  the  14E23  Trainer  and  simulated  SQS-35  Sonar.) 

Finally,  a separate  program  provided  for  data  reduction  and  printout  of 
the  recorded  data.  This  program  used  a computer  located  in  the  Honeywell  Train- 
ing and  Control  Systems  Center,  West  Covina,  California. 

After  the  programs  were  prepared,  a data  collection  plan  was  devised.  Data 
were  to  be  collected  for  ASW  teams  from  six  ships.  Three  of  the  teams  were  to 
use  the  14A2  in  independent  mode,  and  three  were  to  use  the  14A2  and  14E19  in 
joint  mode.  Data  were  to  be  collected  on  three  exercises  per  team,  represent- 
ing three  increasingly  difficult  levels  of  training.  The  training  levels  were 
designated  by  FLEASWTRACENPAC  as  exercise  categories.  Category  1 exercises  were 
intended  for  team  familiarization  with  the  training  equipment  and  overall  team 
training  objectives;  therefore,  data  w- re  not  collected  from  exercises  in  this 
category. 

Category  2 exercises  involved  ownship  (0/S)  and  an  assist  ship,  which  to- 
gether form  a search  attack  unit  (SAU) ; another  ship  designated  as  a high  value 
unit  (HVU) ; and  a target  submarine  (TS) . The  primary  objective  of  the  ASW  team 
was  to  protect  the  HVU  and  prosecute  the  attack  against  the  submarine  with 
rockets  (ASROC)  and  over-the-side  (OTS)  weapons. 

3The  14A2E  is  one  of  two  14A2  Sonar  Trainers  at  FLEASWTRACENPAC. 
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Figure  1.  Simplified  flow  diagram  of  data  collection  programs  for  the  14A2  complex. 


Category  3 exercises  involved  SAU,  HVU,  TS,  and  one  fixed-wing  ASW  aircraft 
(A/C) . The  primary  objective  was  to  protect  the  HVU  and  prosecute  the  attack 
against  the  submarine  with  ASROC,  OTS,  and  A/C  vectored  attacks  (VECTACS) . 

Category  4 exercises  involved  0/S  with  no  assist  ship,  an  HVU,  TS,  and  two 
A/C,  one  fixed-wing  and  one  ASW  helicopter.  The  primary  objectives  were  the 
same  as  for  Category  2;  however,  more  complex  communications  were  involved. 

The  three  categories  of  exercises  were  differentiated  primarily  by  the  com- 
binations of  simulated  vehicles  in  the  exercise  and  by  the  exercise  objectives. 

The  difficulty  level  of  each  exercise  was  reflected  in  more  complex  communica- 
tions; however,  many  operational  tasks  were  not  affected  by  exercise  category. 

An  attempt  was  made  to  establish  as  controlled  a study  as  possible  through 
the  following  conditions: 

1.  Personnel  manning  key  team  positions  were  to  maintain  those  positions 
during  the  three  exercises  used  for  data  collection. 

2.  Instructors  were  to  maintain  the  same  level  of  target  submarine  maneuver- 
ing complexity  and  instructor  assistance  for  all  six  teams  within  a given  exercise 
category. 

3.  Instructors  were  to  use  checklists  specifically  developed  for  this  study 
to  maintain  communications  and  procedural  error  data. 

All  data  were  to  be  collected  at  FLEASWTRACENPAC  during  advanced  ASW  team 
training  (K-2E-1070  course).  All  data  were  to  be  collected  on  the  14A2E  Trainer, 
either  in  independent  mode  or  in  joint  mode  with  the  14E19  (SQS-26CX  Sonar) 
Trainer. 

Data  Reduction 


The  data  were  analyzed  in  terms  of  PPAS  requirements.  The  following  questions 
were  considered: 

1.  What  procedures  would  be  effective  for  summarizing  and  presenting  data 
effectively  to  personnel  managers? 

2.  To  what  degree  does  measured  performance  represent  real  performance? 

3.  What  approaches  would  maximize  the  usefulness  of  information  retrievable 
from  team  training  exercises? 

4.  What  steps  must  be  taken  to  obtain  routinely  the  desired  information 
from  the  14A2  Trainer  Complex?  Specifically,  what  hardware  and  software 
modifications  would  be  necessary  to  implement  an  automated  performance  measure- 
ment system,  and  what  additional  display  or  printout  devices  would  be  needed? 

Problems  Encountered 


Due  to  the  heavy  training  load  imposed  on  FLEASWTRACENPAC,  some  constraints 
were  necessarily  placed  on  the  data  collection  effort.  First,  data  collection 


had  to  be  conducted  on  a "not  to  interfere"  basis.  Second,  instructors  were 
able  to  assist  in  data  collection  only  to  the  degree  that  their  training 
functions  were  not  degraded.  Despite  these  constraints,  data  collection 
was  accomplished  for  all  exercises.  Minor  programming  and  hardware  problems 
were  encountered.  Consequently,  the  following  deviations  from  the  data  col- 
lection plan  resulted: 

1.  Data  were  recorded  for  six  ships  as  planned.  However,  equipment 
configurations  differed  on  some  of  the  ships,  and  this  determined  the  opera- 
tional trainer  setup  used  for  the  ship's  ASW  team. 

2.  Problems  were  experienced  with  some  data  printouts,  and  alternate 
exercise  data  were  used  where  it  was  necessary  and  appropriate. 

3.  As  the  activity  level  increased,  it  was  not  always  possible  for  in- 
structors to  record  procedural  and  communications  errors. 

4.  Since  emphasis  was  placed  on  team  training,  instructors  often  dictated 
sonar  or  UB  plot  equipment  control  settings,  which  reduced  capability  to 
measure  procedural  errors. 

Table  1 shows  the  equipment  configurations  for  the  ships  involved  and  summarizes 
deviations  from  the  original  plan.  These  deviations  did  not  seriously  com- 
promise the  data  analysis  capability,  and  to  some  degree  they  provided  further 
enlightenment  as  to  the  problems  involved  in  using  ASW  team  trainers  to  evaluate 
the  performance  of  sonar  technicians. 


6 


Table  1 

ASW  Teams,  Equipment  Configuration,  and  Deviations 
from  Data  Collection  Plan 


Ship 

Equip.  Configuration 

Deviations 

A 

14A2/14E19  (SQS-26CX  Sonar) 

MK  53  Fire  Control 

None. 

B 

14A2/14E19  (SQS-26CX  Sonar) 

MK  53  Fire  Control 

Data  from  a more  advanced 
used  in  place  of  Exercise 
3. 

exercise 

Category 

c 

14A2/14E19  (SQS-26CX  Sonar) 

MK  53  Fire  Control 

None. 

D 

14A2  (SQS-23  Sonar) 

MK  268  Torpedo  setting  panel 

No  fire  control 

No  fire  control  data. 

No  sonar  accuracy  data. 

E 

14A2/14E23  (SQS-35  Sonar) 

MK  268  Torpedo  setting  panel 

No  fire  control 

Sonar  was  not  SQS-23,  but 
No  fire  control  data. 

No  sonar  accuracy  data. 

similar. 

F 

14A2  (SQS-23  Sonar) 

MK  38  Fire  Control 

No  Exercise  Category  3 data  printout. 
Exercise  Category  4 data  used  for 
two  exercises. 

EVALUATION  OF  POTENTIAL  PERFORMANCE  MEASURES 


All  potential  sonar  and  underwater  battery  (UB)  plot  subteam  performance 
measures  were  evaluated.  Four  categories  of  measures  were  included:  accuracy, 
time,  communications,  and  procedures.  The  basic  criteria  used  to  evaluate 
measures  in  each  category  were  (1)  the  probability  of  obtaining  the  measure 
in  a team  training  exercise,  and  (2)  the  usefulness  of  the  measure  in  a PPAS. 

Sonar  Subteam  Measures 


The  sonar  subteam  is  distinctively  different  for  teams  using  the  SQS-23 
Sonar  and  the  SQS-26CX  Sonar.  The  key  personnel  comprising  the  SQS-23  Sonar 
subteam  are  the  console  operator  and  the  sonar  supervisor.  Key  personnel  for 
the  SQS-26CX  Sonar  are  the  B-scan  (surface  channel)  console  operator,  A-scan 
(deep  channel)  console  operator,  target  tracking  console  (TTC)  operator,  and 
sonar  supervisor. 

Sonar  Accuracy  Measures 

Sonar  Range  and  Bearing  Error.  Sonar  operator  range  and  bearing 
errors  were  calculated  by  comparing  the  ranges  and  bearings  an  operator  obtained 
through  manipulation  of  the  tracking  console  (that  is,  the  observed  positions) 
with  the  actual  ranges  and  bearings  of  the  contacts.  These  data  were  obtained 
by  means  of  specially  developed  computer  programs.  For  the  SQS-23  Sonar  the 
actual  positions  were  available  from  the  past  position  table  in  the  14E19  Main 
Trainer  Program.  The  actual  positions  were  recorded  on  computer  tape  along  with 
the  observed  positions;  the  range  and  bearing  errors  were  calculated  by  means 
of  a computer  program  designed  for  use  at  the  Honeywell  Facility  in  West  Covina. 
For  the  SQS-26  Sonar,  range  and  bearing  errors  were  computed  internally  and  dis- 
played on  the  14E19  instructor's  console.  Error  data  were  obtained  directly 
by  reading  the  range  and  bearing  output  words  in  the  14E19  Main  Trainer  Program. 

Ideally,  range  and  bearing  error  should  be  sampled  once  every  new  data 
time  (NDT) . NDT  is  the  sonar  range  and  bearing  input  to  the  fire  control,  once 
sonar  is  placed  in  contact  and  track  operation.  For  the  SQS-23  Sonar,  NDT  is 
automatic  at  the  end  of  each  sonar  ping  cycle.  For  the  SQS-26,  NDT  is  manually 
entered  using  the  tracking  console  NDT  button,  but  for  all  practical  purposes 
it  is  once  per  sonar  ping  cycle.  For  all  recorded  exercises  the  sonar  range 
scale  was  set  so  that  the  ping  cycle  time  was  12.5  seconds;  consequently,  NDT 
was  12.5  seconds.  Due  to  hardware  limitations  in  this  study,  data  were  sampled 
at  10-second  intervals,  rather  than  each  NDT.  However,  on  two  exercises  it 
was  necessary  to  record  the  range  and  bearing  data  manually,  once  per  NDT,  by 
observing  the  instructor's  console  error  readouts.  A comparison  of  these  two 
exercises  with  all  other  exercises  indicated  that  there  was  no  significant  dis- 
tortion due  to  the  10-second  sampling  interval. 

For  each  ship,  the  arithmetic  average  (X),  standard  deviation  (o),  and 
root  mean  square  (RMS)  for  sonar  range  and  bearing  error  were  calculated.  (See 

Tables  2 and  3.)  The  X represents  a tracking  "bias"  or  average  error, 
while  the  a summarizes  tracking  variance  or  consistency.  The  RMS  provides 
a measure  showing  the  combined  effects  of  tracking  bias  and  consistency, 

where  the  RMS  is  related  to  the  X and  a as  shown  in  the  following  equation: 
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Table  2 


f 


Sonar  Range  Error  (yards) 


Ship 

X 

a 

RMS 

Exercise  Category  2 

A 

103 

46 

113 

B 

98 

93 

134 

C 

144 

80 

164 

F 

227 

273 

348 

Exercise  Category  3 

A 

116 

117 

163 

B 

146 

58 

157 

C 

70 

46 

82 

Exercise  Category  4 

A 

165 

98 

190 

B 

90 

60 

107 

C 

123 

84 

148 

F 

32 

66 

73 

Fa 

67 

80 

103 

g 

For  ship  F there  was  no  exercise  Category  3 data  available;  therefore,  an 
additional  Category  4 exercise  was  recorded. 


10 


RMS 


1/2 


2 , 9 

X + — (o)2 
n 

From  this  equation  it  can  be  seen  that  RMS  increases  if  either  the  X or  a in- 
creases. The  RMS  is  interpreted  as  an  average,  as  is  the  X.  The  X may  appear 
to  show  accurate  tracking  where  positive  and  negative  (leading  and  lagging) 
errors  are  self-canceling,  but  the  alternating  errors,  resulting  from  incon- 
sistency, will  be  evident  in  a larger  a and  a larger  absolute  error  value 
(RMS).  A comparison  between  SHIP  A and  SHIP  B on  range  error  for  Category 
3 (Table  2)  illustrates  this  point. 

Several  constraints  must  be  taken  into  account  when  recording  and 
evaluating  sonar  tracking  error.  First,  the  recording  of  sonar  range  and  bear- 
ing input  should  commence  after  an  established  interval  following  contact,  per- 
haps two  or  three  ping  cycles.  Frequently,  the  first  two  or  three  NDT  inputs 
show  significantly  higher  errors  because  the  operator  goes  from  SEARCH  to 
CONTACT  before  fully  positioning  his  cursor.  The  delay  will  ensure  that  these 
errors  are  not  recorded.  Second,  tracking  error  should  not  be  evaluated  during 
multiple-echo  (M/E)  situations.  Multiple  echoes  occur  when  a simulated  decoy 
vehicle  is  activated  on  the  sonar  displays  at  the  same  range  and  bearing  as 
the  target  submarine.  At  that  time,  the  sonar  and  CIC  teams  are  being  exercised 
in  multiple-echo  procedures  (to  be  explained  in  a subsequent  section)  and  track- 
ing errors  will  be  abnormally  large.  Once  the  operator  decides  which  is  the 
correct  target  and  begins  continuous  tracking,  the  decoy  is  deactivated  on  the 
sonar  display.  At  this  time,  recording  of  error  data  should  be  reinstated  pro- 
vided that  contact  is  maintained.  Finally,  the  automatic  recording  of  tracking 
error  data  should  be  terminated  when  the  sonar  is  switched  to  the  LISTEN  mode, 
out  of  CONTACT  to  LOST  CONTACT,  or  when  the  submarine  is  removed  from  display 
on  sonar  by  instructor  action  or  ensonif ication  math  model  constraints. 

Range  and  bearing  summary  statistics  are  very  descriptive  of  performance 
and  help  pinpoint  basic  tracking  deficiencies  within  a specific  exercise.  The 
instructor  or  trainee  could  determine  if  there  is  specific  tracking  bias  or  in- 
consistency. However,  the  measures  are  probably  not  the  most  appropriate  to 
show  general  tracking  accuracy.  The  tracking  bias  may  be  affected  by  doppler 
and  target  aspect.  The  sonar  operator  may  purposely  position  the  cursor  to  lead 
or  lag  the  target  in  range  depending  on  whether  the  target  is  closing  or  opening 
in  range.  It  was  not  possible  to  fully  assess  this  effect  with  the  limited  data 
base,  and  it  will  probably  be  necessary  to  develop  algorithms  to  attenuate  track- 
ing error  bias  as  a function  of  doppler  rate  and  target  aspect. 

Sonar  Miss  Distance.  Sonar  miss  distance  represents  the  combined  effect 
of  sonar  range  and  bearing  error.  Sonar  miss  distance  can  be  derived  from  range 
and  bearing  error  for  each  NDT.  Obviously,  the  same  constraints  and  limitations 
that  apply  to  collecting  range  and  bearing  error  data  apply  to  sonar  miss  dis- 
tance. 


Sonar  miss  distance  was  calculated  manually  from  sonar  data  printouts. 
Means  and  standard  deviations  are  not  appropriate  measures  of  miss  distance. 
The  arithmetic  mean  is  an  average  of  a set  of  data  along  one  axis,  in  which 
values  of  opposite  direction  are  self-canceling.  The  RMS  is  an  average  of  the 
absolute  values  of  the  miss  distance  without  respect  to  direction.  It  more 
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accurately  reflects  real  error;  consequently,  it  is  a more  appropriate  measure. 
However,  when  the  RMS  miss  distance  is  presented  in  terms  of  total  yards  from 
target,  comparisons  between  ships  cannot  be  readily  made  because  miss  distance 
is  partially  a function  of  target  range.  Therefore,  in  order  to  make  com- 
parisons between  ships  more  meaningful,  the  RMS  total  yards  miss  distances 
were  corrected  for  range  differences  by  calculating  the  RMS  miss  distance  per 
1000  yards  of  range  (RMS  in  mils)  as  shown  by  the  following  formula: 

RMS  miss 

distance  per  = RMS  miss  distance  (total  yards). 

1000  yards  Range/1000 

The  RMS  miss  distance  figures  for  the  four  ships  from  which  tracking 
data  were  collected  are  shown  in  Table  4.  The  miss  distance  RMS  per  1000  yards 
of  range  is  considered  to  be  the  most  appropriate  overall  measure  of  sonar 
tracking  accuracy.  In  conjunction  with  range  and  bearing  error,  sonar  miss 
distance  appears  to  be  a measure  that  should  be  included  in  a PPAS. 

Very  significant  differences  in  tracking  accuracy  were  present  between 
ships  during  initial  exercises;  however,  these  differences  decrease  in  subsequent 
exercises.  It  is  inferred  that  initial  differences  are  indicative  of  differences 
in  experience  or  the  time  interval  since  operators  had  sonar  practice.  Figure  2 
illustrates  the  initial  disparity  in  sonar  tracking  accuracy  and  the  performance 
leveling  achieved  with  practice.  With  very  little  practice  the  differences  were 
reduced.  This  information  could  be  of  consideiable  value  in  determining  how 
often  Fleet  ASW  teams  should  utilize  the  trainers. 

Sonar  Doppler  Error.  The  SQS-26CX  Sonar  used  with  the  14E19  Trainer  can 
be  used  to  measure  sonar  doppler  and  determine  increasing  or  decreasing  range 
rate,  referred  to  as  opening  (down)  or  closing  (up)  doppler.  When  the  target 
tracking  console  (TTC)  operator  positions  the  doppler  cursor  on  the  target 
doppler  indicator  (TDI) , he  is  provided  with  a readout  of  doppler  (range  rate) 
in  knots.  On  the  14E19  Trainer,  doppler  error  is  displayed  on  the  instructor's 
console;  this  error  was  recorded  by  the  14E19  Data  Collection  Program  by  read- 
ing the  doppler  error  computer  output  word  directly  at  10-second  intervals. 

Sonar  doppler  is  input  to  the  fire  control  system  at  each  NDT  and  should 
be  measured  with  the  same  constraints  imposed  in  measuring  bearing  and  range 
error.  An  additional  constraint  is  that  it  should  only  be  measured  when  the 

target  tracking  console  is  in  control  of  either  the  A-scan  or  B-scan.  The  X,  a, 
and  RMS  for  doppler  error  were  calculated  for  each  exercise  in  which  the  14E19 
and  14A2  Trainers  were  run  in  joint  mode.  These  data,  which  are  shown  in  Table  5, 
indicate  that  doppler  error  did  not  decrease  with  practice.  This  suggests  that 
proper  positioning  of  the  TDI  cursor  may  be  a difficult  task  for  the  operator  or 
that  TDI  cursor  positioning  is  not  properly  understood  by  individual  operators. 
This  would  make  doppler  error  an  important  measure  for  application  to  the  PPAS. 

As  with  sonar  tracking  error,  the  X and  a for  doppler  error  are  more  descriptive 
of  specific  performance  deficiencies,  while  the  RMS  provides  a more  appropriate 
overall  measure. 
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Table  4 


100- 


EXERCISE  CATEGORY 


Table  5 


Sonar  Doppler  Error  (knots) 


Exercise  Category  2 


The  SQS-23  Sonar  used  with  the  14A2  Trainer  in  independent  mode  does 
not  have  doppler  tracking  capability.  Doppler  is  recognized  as  an  aural  signal 
and  reported  over  the  61JS  sound-powered  phones  and  29MC  circuit  as  "up"  or 
"down"  doppler.  For  the  SQS-23,  doppler  error  was  considered  to  be  a sonar 
communications  error. 

Sonar  Time  Measures 

Four  time  measures  were  evaluated  for  the  sonar  subteam.  The  four 
measures  are  target  detection  time,  regain  contact  time,  response  to  lost 
contact,  and  time  to  resolve  multiple  echoes  (M/E) . A summary  of  results 
for  the  six  teams  on  which  these  measures  were  obtained  is  presented  in  Table 
6. 

Target  Detection  Time  and  Regain  Contact  Time.  Target  detection 
time  and  regain  contact  time  are  essentially  the  same  measure;  target  detec- 
tion time  refers  to  the  first  time  contact  is  acquired,  while  regain  contact 
time  refers  to  times  when  contact  is  gained  following  lost  contact.  These 
measures  are  not  really  meaningful  under  the  current  circumstances  of  ASW 
team  training;  therefore,  they  are  not  considered  to  be  appropriate  for 
PPAS  use.  (The  current  practice  is  to  keep  the  target  deactivated  on  the 
sonar  display  until  the  target  has  reached  a range  at  which  it  will  appear 
distinctly  when  it  is  activated.) 

Time  to  Resolve  Multiple  Echoes.  Time  to  resolve  multiple  echoes 
appears  to  be  the  most  appropriate  sonar  subteam  time  measure.  Multiple 
echoes  are  simulated  on  the  14A2  Complex  by  activation  of  a second  submarine 
on  the  sonar  displays  at  the  same  position  as  the  target  submarine,  but  dead 
in  the  water.  This  results  in  a display  of  diverging  multiple  echoes,  in  which 
the  second  submarine  represents  a decoy.  Instructors  use  these  echoes  in  two 
ways.  In  the  first,  they  are  generated  until  the  correct  target  is  determined, 
then  the  decoy  is  deleted  from  the  sonar  display.  (The  decoy  may  also  be 
removed  if  the  instructor  decides  the  team  cannot  handle  multiple  echoes  and 
he  wants  the  exercise  to  continue.)  In  the  second,  the  target  submarine  is 
placed  in  a dive  below  sonar  layer  depth  so  that  only  the  decoy  is  displayed. 

In  this  case,  the  team  is  expected  to  determine  that  the  track  is  a decoy  and 
to  go  to  LOST  CONTACT  procedures. 

In  the  multiple  echo  procedures  the  sonar  subteam  immediately  reports 
multiple  echoes  to  CIC.  Then,  the  operator  alternates  between  tracking  the  first 
signal,  designated  "Alpha,"  and  the  second  signal,  designated  "Bravo,"  to 
determine  which  echo  return  is  the  submarine.  (This  determination  is  made 
aurally  through  judgments  of  signal  strength,  echo  quality,  and  doppler.) 

Large  range  and  bearing  errors  are  generated  while  the  operator  is  tracking  the 
decoy.  The  error  magnitudes  increase  and  decrease  as  the  operator  alternates 
between  "Alpha"  and  "Bravo."  At  each  NDT  he  reports  range  and  bearing  for  one 
signal  or  the  other,  as  appropriate.  Upon  resolution,  he  reports  which  track 
is  correct  and  continues  tracking,  or  he  reports  that  he  has  lost  contact  and 
sets  the  sonar  in  SEARCH. 

Time  to  resolve  multiple  echoes  is  currently  determined  by  the  instructors 
by  listening  to  sonar/CIC  communications.  For  this  research,  it  was  determined 
through  an  analysis  of  printouts  of  range  and  bearing  information.  This  informa- 
tion could  be  recorded  automatically  by  having  the  computer  carry  out  a similar 
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Table  6 


Sonar  Subteam  Time  Measures 


Ship 

Target 

Detection 

Time 

(sec) 

Regain 

Contact 

Time 

(sec) 

Resolve 

M/E 

Time 

(sec) 

Response 
to  Lost 
Contact 
(sec) 

Exercise 

Category  2 

A 

34 

21 

Unresolved 

N/A 

B 

30 

47 

N/A 

26/28 

C 

60 

2 

170 

41 

D 

31 

N/Aa 

143 

N/A 

E 

69 

21 

N/A 

N/A 

F 

37 

N/A 

Unresolved 

Didn’t  respond 

Exercise 

Category  3 

A 

72 

45 

Tracked  wrong 

44 

target 

B 

49 

240 

190 

25 

C 

37 

N/A 

185 

24 

D 

19 

N/A 

N/A 

N/A 

E 

57 

N/A 

210 

57 

Exercise 

Category  4 

A 

30 

N/A 

142 

N/A 

B 

19 

46 

90 

70 

C 

50 

N/A 

165 

N/A 

D 

80 

N/A 

170 

N/A 

E 

10 

N/A 

N/A 

N/A 

Fb 

37 

N/A 

N/A 

N/A 

F 

45 

12 

55 

104 

^/A  indicates  that  for 

various  reasons  time 

data  could  not 

be  collected. 

bFor  ship  F there  was  no 

i exercise 

Categorv  3 

data  available 

; therefore,  an 

additional  Category  4 exercise  was  recorded. 
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analysis.  The  start  time  (that  is,  the  time  at  which  the  decoy  is  activated) 
could  easily  be  determined  and  recorded  in  the  trainer’s  computer.  However, 
the  terminate  time  is  more  difficult  to  determine,  although  it  could  be 
established  through  fairly  simple  logic.  If  the  operator's  range  and  bearing 
errors  were  fairly  small  (for  example,  less  than  250  yards  on  the  5000-yard 
scale)  where  the  decoy  was  deactivated,  the  operator  would  be  judged  to  have 
correctly  resolved  the  situation.  The  time  at  which  the  multiple  echoes  were 
resolved  would  be  the  time  at  which  range  and  bearing  errors  were  last  within 
nominal  error  limits  prior  to  decoy  deactivation.  If  errors  were  abnormally 
large  at  the  time  the  decoy  was  deactivated,  the  operator  would  be  judged  to 
have  not  correctly  resolved  the  situation.  To  simulate  a situation  in  which 
the  target  submarine  dives  below  the  sonar  layer,  the  target  submarine  would 
be  deleted  from  the  display  first.  The  time  at  which  the  sonar  is  placed  in 
SEARCH  would  denote  the  time  at  which  the  operator  correctly  determined  that 
he  has  lost  contact.  If  the  decoy  was  deleted  from  the  display  before  the 
sonar  was  placed  in  LOST  CONTACT,  the  operator  would  be  judged  to  have  not 
correctly  resolved  the  situation. 

Table  6 shows  the  results  of  the  multiple-echo  measures.  These  results 
suggest  that  the  sonar  subteam  could  be  evaluated  on  two  counts.  First,  they 
could  be  evaluated  from  a procedural  standpoint,  scoring  positively  if  multiple 
echoes  were  correctly  resolved  and  negatively  if  they  were  not.  Second,  they 
could  be  evaluated  in  terms  of  time  to  resolve  multiple  echoes.  Time  measures 
show  significant  differences  in  the  amount  of  time  taken  to  resolve  multiple 
echoes,  although  the  sample  is  too  small  to  provide  meaningful  statistical  data. 
The  time  measure  reflects  team  efficiency  in  multiple-echo  procedures.  This 
measure  appears  to  be  appropriate  for  use  in  the  proposed  PPAS. 

Response  to  Lost  Contact.  Lost  contact  time  represents  the  inverse  of 
the  target  detection  time  measure.  This  measure,  like  target  detection  time, 
is  probably  not  meaningful  in  the  context  of  team  training  and  therefore  is 
not  suitable  for  PPAS  use.  The  large  number  of  lost  contact  times  missing  from 
Table  6 is  an  indication  of  the  difficulty  of  obtaining  this  measure.  In  some 

exercises  the  target  is  deleted  by  instructor  action.  In  others,  the  target 

is  placed  in  a dive,  and  the  target  signal  is  gradually  attenuated  as  a function 

of  environmental  and  performance  factors.  In  this  case  the  reference  time  can- 

not be  accurately  determined. 

Sonar  Procedures  Measures 

Two  methods  were  used  to  derive  quantitative  measurement  data  on  sonar 
operational  procedures.  First,  sonar  instructors  were  provided  with  sonar  com- 
munications procedures  checklists  that  were  specifically  devised  to  obtain 
quantitative  measures.  Second,  for  the  14E19  and  14A2  joint  exercises,  the 
14E19  Data  Collection  Program  recorded  all  switch  settings  on  the  SQS-26CX 
Sonar  Consoles.  Numerous  difficulties  were  encountered  that  prevented  the 
collection  of  reliable  data;  however,  enough  data  were  taken  to  demonstrate 
that  such  measures  can  be  obtained. 

The  frequency  of  procedural  errors  provides  a direct  measure  of  sonar 
subteam  performance;  however,  it  appears  to  be  infrequently  used.  In  most  cases 
the  instructors  reported  that  "there  were  no  errors  to  be  recorded."  The  errors 
that  were  recorded  indicate  that  several  factors  must  be  considered  in  obtaining 
frequency  measures  of  procedural  errors. 
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First,  ASW  team  training  deemphasizes  basic  operator  skills  and 
emphasizes  the  team's  ability  to  track  the  target,  resolve  M/Es,  and  integrate 
sonar  data  into  a coordinated  ASW  attack.  Consequently,  the  sonar  instructors 
established  initial  equipment  settings  and  allowed  little  variance  in  opera- 
tional procedures  during  the  exercise.  Their  direction  allowed  very  few 
operational  errors  to  be  made. 

Second,  switch  positions  must  be  interpreted  relative  to  tactical 
conditions.  All  the  procedure  errors  that  were  recorded  indicated  a need 
to  do  this.  One  error  involved  the  sonar  display  RANGE  SCALE  control  setting. 
Usually,  the  sonar  scale  is  set  at  XXKYDS,  When  aircraft-vectored  attacks 
(VECTACS)  are  conducted  at  limited  ranges  (but  still  greater  than  the  sonar 
range  scale  setting),  this  establishes  a DATUM  position  for  the  target,  and 
the  sonar  operator  should  shift  to  a longer  range  scale.  In  some  instances, 
operators  failed  to  do  this.  The  second — and  most  commonly  recorded — switch 
positioning  error  was  shifting  too  late  from  SEARCH  and  LOST  CONTACT  to  CON- 
TACT and  TRACK.  The  third  error  of  this  type  was  to  leave  the  sonar  in 
CONTACT/TRACK  once  contact  was  lost. 

In  order  to  provide  automated  recording  of  the  first  type  of  switch 
positioning  error,  software  logic  must  be  devised  to  record  the  VECTAC  position 
and  determine  the  appropriate  range-scale  setting.  For  the  other  errors, 
logic  must  note  the  time  of  activation  or  deactivation  of  the  target  on  the 
sonar  display,  and  the  time  of  LOST  CONTACT /CONTACT  or  SEARCH/TRACK  setting. 

The  time  it  takes  to  get  an  appropriate  setting  can  be  compared  to  criterion 
reference  times  to  determine  if  the  time  of  changing  the  switch  positions  is 
late  and  should  be  registered  as  a procedural  error.  These  examples  of  pro- 
cedural errors  illustrate  the  necessity  for  extensive  programming  to  interpret 
switch  settings  as  part  of  an  automated  performance  measurement  system. 

The  close  directive  control  assumed  by  sonar  instructors  makes  it 
difficult  for  them  to  maintain  fully  reliable  records  of  procedural  errors 
by  recording  them  manually.  This  suggests  that  the  development  of  automated 
recording  of  procedural  errors  would  be  beneficial  in  evaluating  subteam  per- 
formance. It  would  provide  trainees  with  immediate  feedback  and  collect  data 
for  a PPAS.  In  the  PPAS,  automated  recording  would  help  define  those  procedural 
errors  that  occur  most  frequently  and  represent  specific  deficiencies  within 
Fleet  ASW  teams. 

Sonar  Communications  Measures 


Communications  error  data  were  equally  as  difficult  to  obtain  as  pro- 
cedural error  data,  and  relatively  few  errors  were  recorded.  However,  on  two 
exercises  data  were  obtained  for  some  communications  errors,  and  evaluation 
comments  from  instructors  were  acquired  for  several  others.  These  data  suggest 
that  if  a method  could  be  devised  for  automated  recording  of  communications 
errors,  such  as  an  automated  voice-recognition  system,  quantitative  measures 
could  be  developed  that  would  provide  a very  direct  measure  of  performance 
proficiency  for  PPAS  utilization.  Initially,  three  measures  of  communications 
were  proposed: 

1.  Frequency  of  omitted  data  items. 

2.  Frequency  of  late  data  items. 

3.  Frequency  of  Incorrect  data  items. 
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The  frequency  of  incorrect  data  items  does  not  appear  to  be  a relevant  measure. 
Communications  from  sonar  are  usually  concerned  with  reporting  discrete  events, 
for  example,  lost  contact  or  multiple  echoes.  Range  and  bearing  are  reported 
directly  as  obtained  from  console  readouts.  The  operators  do  not  calculate 
values,  such  as  cone-of-courses,  where  considerable  inaccuracies  might  be 
generated.  As  a result,  few  errors  would  be  found  on  this  measure. 

The  frequency  of  omitted  data  items  appears  to  be  the  most  important 
measure.  Specific  reports  for  the  following  items  should  be  transmitted  over 
the  61 JS  or  29MC  circuits: 

1.  Contact/lost  contact. 

2.  Regain  contact. 

3.  Range  and  bearing  at  each  NDT. 

4.  Multiple  echoes. 

5.  No  echoes. 

6.  Hydrophone  effects  (or  torpedo  and  bearing). 

7.  Approaching  baffles. 

8.  Doppler. 

9.  Classification  of  contact. 

The  frequency  of  omitted  or  late  data  items  provides  direct  measures 
of  adherence  to  communications  doctrine. 

UB  Plot  Subteam  Measures 


The  key  personnel  comprising  the  UB  plot  (fire  control)  subteam  are  the 
ASW  fire  control  officer,  attack  plotter  operator,  and  the  ballistics  computer 
operator. 

UB  Plot  Accuracy  Measures 

Target  Motion  Analysis  (TMA)  Course  and  Speed  Error.  Target  motion 
analysis  course  and  speed  data  were  recorded  by  the  14A2  Computer  Data  Col- 
lection program  and  subsequently  analyzed  by  a separate  data  reduction  program. 
The  observed  data  points  for  TMA  errors  were  the  cursor  positions  for  target 
course  and  speed  on  the  attack  plotter  console.  The  attack  plotter  operator 
inserts  course  and  speed  estimates  by  aligning  his  cursors  to  best  fit  the  sonar 
NDT  points  on  the  console.  The  reference  data  points  were  the  actual  course 
and  speed  of  the  target  from  the  main  trainer  program.  Ideally,  course  and 
speed  error  should  be  sampled  following  each  sonar  NDT  input  to  fire  control; 
however,  they  were  recorded  at  10-second  intervals  as  were  the  sonar  data. 

Since  TMA  is  based  on  sonar  input,  it  is  obvious  that  the  conditions 
applying  to  sonar  error  apply  to  TMA  error.  That  is,  recording  should  not 
begin  until  two  or  three  pings  after  contact,  and  TMA  error  should  be  recorded 
while  in  CONTACT  but  not  during  mul _iple-echo  procedures.  Additionally,  course 
or  speed  error  is  not  meaningful  during  target  course  or  speed  maneuvers,  since 
course  or  speed  is  continuously  changing  until  the  maneuver  is  complete. 

In  order  to  obtain  overall  scores  for  TMA  accuracy,  the  X,  o,  and  RMS 
values  were  calculated  for  TMA  course  and  speed  errors;  these  values  are  pre- 
sented in  Tables  7 and  8.  However,  caution  should  be  used  in  interpreting  the 
statistical  values.  Typically,  two  to  four  course  estimates  are  made  between 
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Table  7 


TMA  Course  Error,  (degrees) 


Ship 

X 

a 

RMS 

Exercise  Category  2 

A 

-19.00 

6.37 

20.00 

B 

- 8.32 

13.78 

15.80 

C 

11.88 

5.86 

13.06 

Exercise  Category  3 

A 

- .13 

0 

.13 

B 

3.50 

4.90 

5.68 

C 

-17.28 

8.30 

18.99 

Exercise  Category  4 

A 

-10.05 

0 

10.05 

B 

.23 

0 

. 23a 

(6.56) 

(0) 

(6.56) 

C 

7.95 

9.42 

11.72 

F 

- 4.46 

9.62 

8.13 

F 

12.06 

10.88 

15.96 

Note. 

Ships 

teams 

D and  E did 
did  not  use 

not  havi  fire  control  systems;  consequently, 
that  equipment  in  the  trainer. 

the 

The  first  value  is  for 
(shown  in  parentheses) 

estimate  prior  to  submarine  maneuver,  and  the 
is  for  estimate  after  maneuver. 

second 

Table  8 


* 


TMA  Speed  Error  (knots) 


Ship 

X 

a 

RMS 

Exercise  Category  2 

A 

7.37 

.13 

7.40 

B 

- 3.21 

2.86 

4.25 

F 

3.88 

2.06 

4.33 

Exercise  Category  3 

A 

- 2.13 

0 

2.13 

B 

-12.54 

.65 

12.56 

C 

-11.63 

.04 

11.63 

Exercise  Category  4 

A 

10.09 

0 

10.09 

B 

2.91 

0 

2.91 

C 

-12.68 

.89 

12.71 

F 

5.97 

0 

5.97 

F 

4.24 

9.00 

9.60 

Note. 

Ships 

teams 

D and  E did 
did  not  use 

not  have  fire  control  systems;  consequently, 
that  equipment  in  the  trainer. 

the 

the  time  of  contact  and  the  first  weapon  firing,  with  the  first  adjustment 
occurring  about  30  to  60  seconds  after  contact.  Occasionally,  an  original 
estimate  may  be  set  based  on  a CIC  estimate,  and  no  readjustment  will  be 
made  following  contact  if  the  TMA  course  cursor  appears  to  be  a good  fit  to 
the  subsequent  NDT  points.  This  is  very  likely  if  there  is  little  time  be- 
tween contact  and  weapon  firing  or  very  inconsistent  NDT  inputs  to  fire  con- 
trol. In  this  case  the  a would  equal  zero,  but  it  would  not  necessarily 
reflect  good  performance  by  the  attack  plotter  operator.  One  operator's 
errors  on  initial  TMA  may  be  considerably  reduced  by  time  of  fire,  which  is 

desirable;  still,  a large  X would  result.  The  X of  another  operator  may  be 
smaller  but  only  reflect  self-canceling  errors.  For  this  reason  the  RMS, 
in  addition  to  indicating  TMA  at  time  of  fire,  is  helpful  in  showing  the  over- 
all effect  of  TMA  estimates.  For  diagnostic  information,  a generated  plot  of 
TMA  estimates  and  actual  submarine  positions  would  show  specific  points  where 

course  and  speed  estimates  improved  or  degraded.  The  X and  a values  for  course 
and  speed  errors  should  be  used  in  conjunction  with  TMA  error  plots  to  ensure 

appropriate  interpretation  of  the  values.  Additionally,  the  X and  a should  be 
compared  to  the  course  and  speed  at  time  of  fire.  As  with  sonar  bearing  and 
range  error,  TMA  course  and  speed  error  can  be  converted  to  miss  distance, 
which  provides  a more  comprehensive  or  generalized  measure. 

TMA  Miss  Distance.  TMA  miss  distance  represents  the  combined  effect 
of  TMA  course  and  speed  error.  It  is  calculated  from  generalized  equations 
along  with  the  sonar  tracking  miss  distance.  TMA  miss  distance  could  be  used 
in  place  of  course  RMS  and  speed  RMS  to  provide  a single  TMA  (UB  plot  subteam) 
accuracy  score.  Data  for  TMA  miss  distance  are  provided  in  Table  9. 

The  very  significant  differences  (j>  < .01)  in  TMA  accuracy  sampled  in 
course  and  speed  errors  and  in  miss  distance  cannot  be  attributed  to  attack 
plotter  operator  performance.  The  ability  to  estimate  target  course  and  speed 
is  directly  related  to  the  variability  of  sonar  range  and  bearing  inputs. 

The  greater  the  variability  in  sonar  tracking  accuracy,  the  more  difficult  it 
is  for  the  plotter  operator  to  estimate  target  course  and  speed,  although  the 
exact  relationship  between  these  two  measures  has  not  been  established.  A 
procedure  must  be  developed  for  converting  IMA  course,  speed,  and  miss  dis- 
tance scores  to  scores  that  are  adjusted  for  variance  in  sonar  inputs.  This 
procedure  might  be  based  on  the  difference  between  the  TMA  solution  and  a 
least  squares  solution  to  sonar  NDT  data  points  on  the  UB  plot  attack  plotter. 

Manual  Offsets  Error.  Setting  the  TMA  manual  offsets  for  an  ASROC  attack 
is  an  important  function  of  the  ballistics  computer  operator  at  the  MK38/53  Fire 
Control  Console.  The  operator  inserts  a manual  bearing  offset  and  manual  range 
offset  that  together  offset  the  weapon  entry  point  (WEP)  generated  by  fire 
control.  The  offset  values  correspond  to  distance  in  yards.  The  operator  in- 
serts the  manual  offsets  just  before  the  weapon  is  fired.  He  does  this  for  two 
reasons.  First,  the  submarine  may  make  a last-second  maneuver  just  before  the 
attack.  The  manual  offsets  allow  the  operator  to  correct  for  this  maneuver. 
Second,  the  torpedo  has  a specific  search  pattern  that  controls  its  direction 
after  it  is  activated  at  the  preset  search  depth.  The  operator  may  insert 
manual  offsets  to  account  for  the  search  pattern  so  that  the  torpedo  will  be 
more  effectively  directed  towards  the  submarine.  The  offset  values  are  directly 
dependent  on  the  problem  geometry.  The  computer-generated  WEP  does  not  account 
for  the  torpedo  search  pattern. 
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Table  9 


TMA  Miss  Distance  Averages 


Ship 

RMS 

(yards) 

RMS 

(mils) 

Exercise  Category 

2 

A 

356.6 

37.5 

C 

236.7 

26.6 

F 

167.6 

25.8 

Exercise  Category 

3 

A 

70.0 

9.7 

B 

507.0 

53.7 

C 

444.9 

50.0 

Exercise  Category 

4 

A 

416.0 

46.0 

B 

115.0 

15.5 

C 

486.9 

54.1 

F 

182.4 

33.2 

F 

141.6 

26.6 

Note. 

Ships  D and  E did  not  have  fire 
consequently,  the  teams  did  not 

control  systems; 
use  that  equipment 

in  the  trainer. 
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Algorithms  can  be  developed  to  compute  the  ideal  offsets  when  the 
manual  offset  controls  are  set;  these  algorithms  would  be  based  on  the  firing 
bearing,  target  course  and  speed,  and  the  torpedo  search  pattern.  Comparison 
of  the  actual  settings  to  the  ideal  settings  would  provide  measures  of  manual 
bearing  and  range  offset  errors  for  a given  attack.  These  measures  could  be 
converted  to  a total  manual  offset  miss  distance  error.  This  overall  measure 
would  provide  a second  accuracy  measure  that,  along  with  TMA  miss  distance, 
would  be  used  to  evaluate  fire  control  operators.  Total  manual  offset  miss 
distance  and  error  would  provide  an  appropriate  measure  for  PPAS  utilization. 

UB  Plot  Time  Measures 

Two  UB  plot  time  measures  were  considered,  TMA  solution  time  and  TMA 
adjustment  time.  These  measures  reflect  essentially  the  same  performance 
parameter  under  different  conditions.  Therefore,  they  are  discussed  separately. 

A summary  of  time  measures  is  presented  in  Table  10. 

TMA  Solution  Time.  TMA  solution  time  measures  the  amount  of  time  re- 
quired for  the  attack  plotter  operator  to  establish  an  initial  solution.  For 
this  study,  "start"  was  the  initial  contact  time.  "Terminate"  was  the  time 
at  which  the  TMA  course  and  speed  cursor  were  set  to  ensure  that  course  error 
was  20  degrees  or  less,  and  course  and  speed  error  remained  stabilized  for 
more  than  10  seconds  or  for  two  NDT  data  points. 

Ability  to  obtain  TMA  solution  time  was  limited  by  two  conditions.  First, 
the  target  could  not  be  maneuvering  immediately  before  the  initial  solution.  In 
this  event  the  measure  was  voided.  Second,  TMA  course  error  could  not  be  < 20 
degrees  at  the  time  of  contact.  As  noted  in  the  discussion  of  TMA  course  and 
speed  error,  the  operator  can  set  his  cursors  based  on  CIC  initial  estimates  of 
target  course  and  speed  in  preparation  for  a rapid  TMA  solution  once  contact  is 
obtained.  The  terminate  time  will  be  the  time  of  the  first  stabilized  cursor 
adjustment  following  contact. 

TMA  Readjustment  Time.  TMA  readjustment  time  is  the  equivalent  of 
TMA  solution  time  following  a target  maneuver  or  multiple  echoes.  The  start 
time  was  either  the  time  at  which  the  target  completed  its  maneuver  or  the 
time  at  which  the  decoy  was  deleted  from  the  sonar  displays.  The  observed  time 
was  established  through  the  same  set  of  criteria  used  for  TMA  solution  time.  In 
any  single  exercise,  several  measures  of  TMA  solution  or  readjustment  time  might 
be  obtained,  due  to  several  attacks  or  target  maneuvers.  In  this  case,  the 
appropriate  value  for  providing  a time  measure  of  UB  plot  subteam  performance 
would  be  the  arithmetic  average  of  the  measures.  The  term  "TMA  convergence 
time"  will  be  used  to  refer  to  both  TMA  solution  and  TMA  adjustment  times. 

Average  TMA  convergence  time  provides  a good  measure  for  PPAS  use. 

UB  Plot  Communications  Measures 

The  same  difficulties  that  were  experienced  in  collecting  sonar  com- 
munications data  were  encountered  in  UB  plot;  consequently,  almost  no  com- 
munications error  data  were  collected.  The  frequency  of  omitted  or  late  data 
items  could  provide  valuable  inputs  to  PPAS.  However,  it  is  almost  impossible 
to  obtain  these  measures  until  a system  is  developed  for  automatically  record- 
ing them.  When  a system  is  developed,  the  following  items  of  UB  plot  to  CIC 
communication  information  should  be  monitored: 


* 


26 


F 1 


Table  10 

UB  Plot  Subteam  Time  Measures 


Ship 

2 

Exercise  Category 

3 

4 

TMA  Solution  Time  (seconds) 

A 

139 

51 

57 

B 

N/Aa 

73 

59 

C 

164 

164 

78 

F 

80 

N/A 

40 

TMA  Adjustment  Time  (seconds) 


A 

N/A 

N/A 

56 

B 

N/A 

74 

85 

C 

78 

54 

64 

F 

149 

27 

70 

aN/A  indicates  that  for  various  reasons  the  time  measures  were  not  available. 
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1.  Report  TMA  solution  (with  each  update). 

2.  Report  weapon  selection. 

3.  Report  gyro  selection. 

4.  Request  permission  to  fire. 

5.  Report  standby. 

6.  Report  weapon  fire  away. 

7.  Request  director  (request  the  fire  control  director  radar  to 
track  an  ASROC  shot  and  obtain  an  accurate  fix  on  the  actual 
WEP  point). 

UB  Plot  Procedures  Measures 

During  the  entire  group  of  exercises,  only  one  UB  plot  error  was  noted 
by  the  instructors,  although  it  was  noted  on  three  exercises:  the  operator 
failed  to  use  the  director  control  function.  This  function  provides  fire 
control-aided  tracking,  which  means  that  it  directs  the  TMA  solution  course 
and  speed  back  to  the  sonar  to  assist  in  tracking.  A measure  of  procedural 
errors  would  be  significantly  more  critical  in  more  complex  sonar  exercises 
involving  bottom  bounce  or  convergence  zone  modes  of  operation.  Data  should 
be  collected  on  such  exercises  to  determine  the  degree  to  which  procedural 
errors  increase. 

On  the  basis  of  the  instructors'  observations,  it  would  appear  that  UB 
plot  operational  errors  rarely  occur;  however,  an  evaluation  of  exercise  data 
indicates  that  this  is  not  necessarily  so.  For  example,  the  TMA  solution  is 
greatly  degraded  by  multiple  echo  procedures  conducted  by  the  sonar  subteam. 

When  multiple  echo  procedures  are  being  conducted,  the  attack  plotter  operator 
should  place  the  fire  control  system  in  the  position  keeping  (PK)  mode.  In 
this  mode,  the  ballistic  solution  is  maintained,  and  new  sonar  range  and  bearing 
data  are  not  accepted.  When  multiple  echoes  are  resolved,  the  UB  plot  console 
should  be  reset  to  the  attack  mode  to  receive  sonar  data.  On  some  exercises, 
the  operator  failed  to  follow  this  procedure,  and  a large  TMA  error  resulted. 

In  others,  the  ballistics  computer  operator  failed  to  insert  appropriate  beat- 
ing and  range  manual  offsets.  As  with  the  sonar  subteam,  UB  plot  subteam  pro- 
cedural errors  could  be  recorded  through  automatic  switch  monitoring.  A count 
of  the  relative  frequencies  of  procedural  errors  should  be  very  helpful  in 
diagnosing  the  specific  types  of  errors  that  are  commonly  committed  by  the  ASW 
teams.  This  is  the  type  of  information  that  should  be  collected  by  a PPAS. 


USE  OF  STG  MEASUREMENT  DATA  IN  A PERFORMANCE 
PROFICIENCY  ASSESSMENT  SYSTEM 


Applicable  STG  Performance  Measure 

Evaluation  of  the  individual  performance  measures  for  the  sonar  and  UR 
plot  subteams  delineated  in  the  preceeding  section  shows  that  the  following 
STG-related  measures  would  be  applicable  for  PPAS  use: 

1.  Sonar  Billet  Measures 

a.  Sonar  tracking  range  error. 

b.  Sonar  tracking  bearing  error. 

c.  Sonar  tracking  miss  distance. 

d.  Sonar  doppler  error. 

e.  Time  to  resolve  M/E. 

f.  Frequency  of  communications  errors. 

g.  Frequency  of  procedural  errors. 

2.  UB  Plot  Billet  Measures 

a.  TMA  course  error. 

b.  TMA  speed  error. 

c.  TMA  miss  distance. 

d.  Manual  offsets  miss  distance. 

e.  TMA  convergence  time. 

f.  Frequency  of  communications  errors. 

g.  Frequency  of  procedural  errors. 

Major  Factors  Limiting  Use  of  the  STG  Data 

Several  factors  limit  the  use  of  the  performance  measurement  data  that 
are  available  from  ASW  team  exercises.  First,  the  data  are  not  comprehensive. 
In  the  evaluation  of  the  individual  measures,  it  was  noted  that  these  measures 
may  provide  "indices"  of  STG  performance  proficiency,  but  they  do  not 
show  all  aspects  of  performance. 

Second,  the  manner  in  which  the  14A2  ASW  Team  Trainer  is  used  by  FLEASWTRA- 
CENPAC  and  the  ASW  Fleet  must  be  taken  into  account.  Currently,  all  of  the 
teams  assigned  to  ASW  surface  ships  are  required  to  undergo  training  at  regular 
intervals.  Their  Commanding  Officers  can  require  additional  training  if  they 
decide  it's  needed.  Normally,  the  ship's  officers  decide  which  team  members 
will  man  the  various  operator  stations.  The  usual  practice  is  to  rotate  the 
team  members  among  the  two  or  three  positions  that  they  might  assume  at  sea, 
to  ensure  that  each  individual  on  the  team  receives  all  relevant  practice. 
Additionally,  personnel  are  frequently  reassigned  to  a new  ship  ox  position. 
These  factors  could  create  a problem  in  attempting  to  obtain  representative 
samples  for  Fleet  evaluation.  Subroutines  must  be  incorporated  within  the 
various  procedures  for  data  summarization  that  account  for  some  ships  or  in- 
dividuals having  repeated  training  sessions,  while  others  have  relatively  few 

Third,  some  measures  cannot  always  be  attributed  to  a single  individual  or 
station.  Good  examples  of  this  are  found  in  counting  procedural  errors  at 
the  sonar  or  UB  plot  consoles.  Usually,  the  sonar  supervisor  and  Firing  Petty 


Officer  (FPO)  exercise  close  direction  over  the  individual  console  operators, 
especially  with  respect  to  control  settings;  consequently,  it  usually  is  not 
possible  to  determine  who  has  committed  a control  setting  error.  In  this 
study,  all  procedural  errors  were  attributed  to  the  individual  console  operators 
and  to  the  sonar  supervisor  or  FPO,  as  appropriate. 

Finally,  it  was  assumed  that  the  variables  that  tend  to  enhance  or  diminish 
certain  aspects  of  performance  in  ASW  exercises  (for  example,  problem  geometry 
and  tactics)  would  tend  to  be  self-canceling  when  averaged  over  a large  enough 
sample,  and  that  this  would  result  in  representative  measures  of  performance. 
Consequently,  while  the  performance  measures  can  be  extracted  for  use  in  a PPAS 
to  provide  generalized  indices  of  Fleet  proficiency,  they  cannot  provide  a 
comprehensive  evaluation  of  ai  individual  exercise  or  ship.  Thus,  data  for 
an  individual  ship  can  only  be  fully  understood  in  the  context  of  all  the  con- 
ditions that  prevailed  for  each  exercise.  As  a result,  the  performance  measure- 
ment data  will  show  specific  areas  of  team  weaknesses  or  strengths  and  identify 
specific  deficiencies  at  the  Fleet  level;  but  by  themselves  they  cannot  identify 
causes  for  these  deficiencies. 

Procedures  for  PFAS  Data  Summarization,  Printout,  and  Evaluation 

The  data  to  be  summarized  and  printed  out  for  use  with  a PPAS  are  described 
in  two  categories:  (1)  detailed  level  data  and  (2)  summary  level  data.  The 
ability  to  obtain  these  data  depends  upon  implementation  of  a comprehensive  per- 
formance measurement  and  data  collection  system  within  the  ASW  team  trainer  to 
monitor  team,  subteam,  and  individual  actions  and  store  all  data  for  reduction, 
evaluation,  and  printout  after  the  exercise  is  completed.  The  following  dis- 
cussion assumes  that  the  specific  hardware  and  software  requirements  to  collect 
all  such  data  can  be  met. 


Preparation  and  Printout  of  Detailed  Data 


The  first  stage  of  data  reduction  for  PPAS  use  is  to  obtain  detailed  level- 
of-performance  measures  for  each  individual  and  ship.  Figures  3 and  4 show 
examples  of  (hypothetical)  detailed  level  statistics  for  the  sonar  and  UB  plot 
subteams.  Note  that  the  data  are  referenced  in  the  14E19  (SQS-26CX  Sonar) 
Trainer  or  MK53  Console  of  the  UB  plot  fire  control  system,  as  appropriate. 
Detailed  level  and  resultant  summary  data  should  be  maintained  separately,  and 
separate  data  printouts  and  memory  storage  should  be  established  for  each  sonar 
trainer  (for  example,  14E23  or  14E24)  and  the  MK  38  Fire  Control  in  a manner 
analogous  to  that  shown  for  the  14E19  Trainer  and  MK  53  Fire  Control  console. 


Each  performance  measure  should  be  attributed  to  a specific  individual  depend- 
ing on  the  billet  positions  that  are  manned  during  the  exercise.  For  example, 
the  RMS  doppler  error  and  RMS  sonar  track  miss  distance  are  assigned  to  either 
the  B-scan  operator  or  the  A-scan  operator,  depending  on  who  was  in  CONTACT  and 
tracking  the  target  submarine.  Some  measures  are  assigned  to  two  individuals. 

For  example,  the  time,  procedures,  and  communications  measures  for  the  sonar 
subteam  are  assigned  to  both  the  console  operator  and  the  sonar  supervisor. 
Although  the  errors  are  made  at  the  individual  consoles,  the  sonar  supervisor 
has  direct  control  of  operator  actions  affecting  these  three  measures,  and  he 
is,  therefore,  at  least  partially  responsible  for  the  errors.  The  values  for 
the  sonar  supervisor  appear  in  parentheses  in  Figure  3 to  indicate  that  they 
are  being  noted  twice  but  not  summed  twice.  The  sum  of  all  the  scores  for  the 
console  operators  is  scored  against  the  sonar  supervisor. 
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Abbreviations:  SPR  - Supervisor,  A - A-scan  Operator,  B - B-scan  Operator,  TIC  - Target  Tracking  Console  Operator 


The  second  stage  of  data  reduction  is  to  summarize  data  at  individual 
and  ship  levels  across  all  exercises.  Figure  5 shows  an  example  of  ship- 
level  statistical  data  for  the  sonar  billets.  The  data  for  UB  plot  billets 
are  summarized  in  the  same  manner.  In  this  summation,  data  are  divided  into 
two  categories;  measures  for  the  first  exercise  for  each  individual,  and 
measures  for  the  last  exercise  (or  average  of  the  last  n_  exercises)  for  each 
individual.  This  refers  specifically  to  the  first  and  last  exercises  in  which 
the  individual  is  at  a certain  position  in  which  a given  measure  is  taken. 

For  example,  a comparison  of  Figures  3 and  5 shows  that  the  first  and  last 
RMS  doppler  error  scores  attributed  to  John  Doe  4 correspond  to  the  first  and 
last  exercises  in  which  he  occupied  the  TTC  console  operator’s  position.  The 
data  from  the  first  exercises  represent  pretraining  proficiency — that  is,  the 
level  of  performance  that  has  been  maintained  since  the  last  training  was 
received.  The  data  from  the  last  exercises  represent  the  proficiency  level 
that  is  achieved  at  the  end  of  ASW  team  training.  In  the  discussion  of  sonar 
tracking  miss  distance,  it  was  shown  that  pretraining  and  posttraining  scores 
may  be  significantly  different. 

Once  scores  are  categorized  into  pretraining  and  post  training  groups,  they 
can  be  averaged  to  represent  ship  scores  in  each  category.  For  time,  procedures, 
and  communications  measures,  the  ship's  score  would  be  the  simple  arithmetic 

average  (X).  Measures  that  are  RMS  values  must  use  a slightly  different  pro- 
cedure. Where  there  are  several  estimates  of  RMS  for  the  same  individual,  such 
as  RMS  for  each  exercise,  or  several  independent  values  for  each  ship,  the  best 

2 

estimate  of  the  team  value  is  the  pooled  variance  (Sp  ) (Dixon  & Massey,  1957). 
For  this  particular  application,  the  pooled  variance  is  the  pooled  RMS  (RMS  ), 
which  is  obtained  as  follows: 


S 

P 
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An  example  of  this  is  shown  by  average  pretraining  sonar  miss  distance  for 
ship  1 in  Figure  5,  where 


(57. 6)2  + (88. 6)2 

2 

Regardless  of  the  procedure  (X  or  RMS)  used  for  each  measure,  the  user  of 
these  data  (the  Navy  personnel  manager)  may  interpret  the  resultant  value  as 
simply  a ship  average.  The  ship  averages  can  then  be  averaged  to  obtain  a total 
sample  average.  For  purposes  of  illustration,  it  is  assumed  that  data  would  be 
summarized  over  three-month  periods.  At  the  FLEASWTRACENPAC , approximately  25 
ships  go  through  the  trainers  each  quarter.  Quarterly  averages  would  be  obtained 
in  the  same  manner  as  ship  averages,  using  ships'  scores  as  individual  samples. 
The  quarterly  average  for  pretraining  proficiency  of  sonar  miss  distance,  for 
example,  would  be  obtained  as  follows: 


1/2 

= 74.7. 
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gure  5.  Hypothetical  ship  level  proficiency  data  for  sonar  billets 
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The  detailed  level  statistics  and  ship  level  statistics  could  be  provided 
as  data  printouts  in  the  formats  shown  in  Figures  3 to  5.  If  these  data  are  to 
be  available  for  the  PPAS,  they  will  have  to  be  generated  through  post-exercise 
data  reduction  and  stored  on  disk  or  in  memory  core  at  the  levels  of  reduction 
shown.  The  data  would  then  have  to  be  transferred  to  tape  and  retrieved  on  a 
periodic  basis  from  the  various  team  trainer  locations. 

Use  of  Detailed  Level  Data  in  the  PPAS 

The  detailed  level  data  for  individuals  and  for  ships  are  intended  for 
use  in  providing  appropriate  information  concerning  specific  training  deficiencies 
as  they  relate  to  (1)  personnel  assignment,  selection,  and  utilization;  and  (2) 
shipboard  and  school  training  of  STG  personnel.  The  data  obtained  from  ASW 
team  training  exercises  cannot  in  themselves  provide  this  information.  They 
must  be  collected  on  tape  and  integrated  into  a more  extensive,  central  PPAS 
data  base.  The  social  security  number  of  each  individual  provides  the  means 
by  which  the  PPAS  can  obtain  specific  demographic  data,  personnel  assignments, 
length  of  service,  type  and  extent  of  training,  specific  courses  taken,  achieve- 
ment scores,  or  other  information.  Relationships  between  various  ASW  team  pro- 
ficiency measures  and  numerous  variables  involved  in  training,  personnel  assign- 
ment, selection,  and  utilization  can  be  established  through  statistical  pro- 
cedures such  as  analysis  of  variance,  correlation  procedures,  and  time  series 
analysis.  These  procedures  cannot  be  defined  within  the  scope  of  this  effort, 
since  the  specific  types  of  demographic  data  that  would  be  available  to  a PPAS 
data  base  have  not  yet  been  defined. 

Preparation  and  Printout  of  Summary  Data 

Summary  data  should  be  partitioned  into  two  groups,  one  for  performance 
proficiency  and  the  other  for  diagnostic  deficiency.  Suggested  performance 
proficiency  data  printouts  for  sonar  and  UB  plot  subteam  billets  are  shown  in 
Figures  6 and  7.  Like  the  ship  level  statistics  data,  these  data  are  divided 
into  pretraining  and  posttraining  proficiency  data.  Comparison  of  Figures  6 and 
7 shows  that  the  quarter  averages  for  the  various  measures  are  taken  directly 
from  the  ship  level  statistics,  as  previously  described.  Quarterly  averages 
from  the  immediately  preceding  quarter  should  be  provided  to  permit  a direct 
comparison  of  changes  in  proficiency  levels.  During  any  given  quarter,  a 
relatively  small  sample  of  Fleet  ships  may  receive  training,  and  these  ships 
may  not  be  fully  representative  of  general  Fleet  proficiency  levels.  To  account 
for  this,  data  should  be  summarized  over  the  last  three  or  four  quarters  and 
presented  as  long-term  averages.  As  with  the  quarterly  averages,  the  long- 
term averages  should  be  updated  on  a quarterly  basis.  Providing  both  quarterly 
and  long-term  averages  allows  a general  overview  of  both  short-term  effects  and 
long-term  trends  in  proficiency  for  the  various  performance  areas.  Along  with 
the  averages  for  the  several  measures,  the  average  time  (months)  between  exercise 
training  sessions  for  ships'  teams  should  be  shown.  This  information  may  be 
helpful  in  determining  the  level  of  training  (amount  of  training,  time  between 
training  sessions,  or  training  complexity)  that  is  appropriate  in  maintaining 
desired  proficiency  levels  throughout  the  Fleet. 
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The  summary  printouts  provide  indices  of  performance;  however,  they  do 
not  provide  diagnostic  data  that  identify  specific  deficiencies.  Although 
the  accuracy  and  time  measures  are  very  exact,  the  frequency  measures  for 
communications  and  procedures  are  strictly  generalized  summaries.  For  this 
reason,  diagnostic  deficiency  data  printouts  containing  the  information 
presented  in  Figures  8 to  11  should  be  used  to  augment  the  summary  proficiency 
data.  Specific  errors  are  shown  for  the  sonar  and  UB  plot  subteams  in  com- 
munications and  operational  procedures.  A separate  printout  is  provided  for 
each  type  of  error. 

Use  of  Summary  Level  Data  in  the  PPAS 

The  PPAS  summary  level  data  printouts  are  intended  to  provide  the  Navy 
personnel  manager  with  an  overview  of  performance  proficiency  levels  and  specific 
types  of  deficiencies  among  STG  personnel.  These  printouts  are  designed  to 
enhance  quality  control.  This  is  the  primary  objective  of  the  PPAS,  On  the 
Fleet  summary  data  printouts  (Figures  6 and  7),  the  division  of  scores  into 
pretraining  and  posttraining  groups  provides  an  overview  of  the  proficiency 
levels  maintained  by  personnel  while  at  sea  and  the  increase  in  proficiency 
levels  resulting  from  exercises  in  the  team  trainer.  This  information  will 
help  managers  identify  areas  in  which  performance  levels  are  unsatisfactory. 

The  presentation  of  averages  for  both  current  and  past  quarters  as  well 
as  long-term  averages  is  intended  specifically  to  reflect  the  actions  implemented 
by  various  Navy  managers  in  controlling  the  Navy  training  process.  The  comparison 
of  current  and  long-term  averages  provides  indices  of  general  trends  in  pro- 
ficiency in  specific  areas.  However,  since  the  long-term  averages  are  generated 
for  relatively  large  samples  of  ships,  they  are  generally  not  very  sensitive  to 
changes  in  personnel  processes.  For  example,  the  benefits  of  improved  training 
methods  could  not  be  observed  until  personnel  who  have  been  trained  by  them 
reach  the  Fleet  and  begin  to  use  the  team  trainers  regularly  in  large  numbers. 

By  comparison,  the  quarter  averages  generated  from  smaller  samples  of  ships 
may  not  reflect  Fleet  levels  of  performance  as  accurately  but,  since  they  are 
based  on  completely  updated  data,  are  far  more  sensitive  to  the  immediate  effects 
of  change. 

The  diagnostic  deficiency  data  printouts  (Figures  8 to  11)  provide  more 
detailed  information  on  the  communications  and  procedures  frequency  measures 
presented  in  the  Fleet  summary  data.  These  printouts  should  enable  measures  to 
identify  specific  errors  that  are  commonly  committed  across  all  teams.  This 
informatic  should  provide  a basis  for  more  informed  decisions  about  changes  in 
ASW  team  procedures.  In  general,  the  suggested  data  printouts  should  meet  the 
objectives  of  the  PPAS  concept  by  providing  proficiency  data  that  is  readily 
understood  by  various  Navy  decision  makers. 
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Diagnostic  deficiency  data  for  sonar  communications 


Communications  Errors 


Figure  9.  Diagnostic  deficiency  data  for  UB  plot  communications 


Procedural  Errors  in  Control  Settings 
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Figure  10.  Summary  printout  of  diagnostic  deficiency  data  for  sonar  procedures. 


Procedural  Errors  in  Control  Settings 


Figure  11.  Diagnostic  deficiency  data  for  UB  plot  procedures. 


DEVELOPMENT  OF  A DATA  COLLECTION  SYSTEM  FOR  THE  PPAS 
General  Requirements  for  a DCS 

An  evaluation  of  the  potential  performance  measures  and  the  ASW  team 
training  objectives  and  requirements  has  highlighted  a number  of  general 
requirements  for  a PPAS  Data  Collection  System  (DCS): 

1.  The  DCS  must  collect  data  continuously  at  real-time  and  store  them 
for  reduction  after  the  exercise  is  completed.  These  data  include  vehicle 
position  readings,  equipment  control  settings,  and  sonar  and  fire-control 
cursor  positions  and  cursor  errors. 

2.  The  DCS  must  provide  real-time  processing  or  post-exercise  batch  pro- 
cessing to  convert  data  to  usable  form  (for  example,  sonar  range  and  bearing 
error  converted  to  sonar  miss  distance;  TMA  course  and  speed  error  converted 
to  TMA  miss  distance). 

3.  The  DCS  must  include  an  executive  routine  that  monitors  overall  trainer 
processing  and  controls  activation  or  deactivation  of  data  collection  for  each 
performance  measure. 

4.  The  DCS  must  provide  a means  of  collecting  and  storing  data  that  identifies 
all  STG  personnel  and  their  billet  stations  for  each  exercise.  The  system  must 
be  capable  of  relating  individuals  to  the  various  PPAS  performance  measures. 

5.  The  DCS  must  retrieve,  reduce,  and  store  data  for  each  individual  and 
each  ship.  These  data  must  be  stored  on  tape. 

6.  The  taped  data  must  be  integrated  into  a larger  PPAS  data  base  that  is 
used  for  summarization  and  printout  of  Fleet  summary  proficiency/deficiency  data 
and  for  statistical  analyses  relating  detailen  measurement  data  to  appropriate 
personnel  training,  selection,  and  assignment  variables. 

7.  The  DCS  must  collect  data  from  various  models  of  the  14A2  Trainer. 

8.  The  DCS  must  collect  data  from  the  14A2  Trainer  in  joint  mode  with 
other  sonar  trainers  (for  example,  14E19  and  14E23)  as  well  as  the  independent 
mode . 

These  requirements  provide  a partially  automated  PPAS  DCS.  This  system  would 
collect  and  reduce  all  accuracy  and  time  data  as  well  as  personnel  data.  However, 
performance  measurement  data  for  communication  errors  and  procedural  errors  would 
require  instructor  monitoring,  evaluation,  and  recording.  Data  checkoff  lists 
could  be  used  for  these  tasks.  The  diagnostic  deficiency  data  sheets  shown  in 
Figures  8 to  11  are  presented  as  formats  for  computer-generated  printouts;  how- 
ever, checkoff  lists  with  similar  formats  could  be  used  for  manual  recording  of 
errors.  These  forms  would  show  data  recorded  for  each  exercise,  instead  of  f.  r 
each  ship.  As  data  collection  was  completed  for  each  ship,  the  data  would  be 
summarized  and  transferred  to  forms  identical  to  those  shown  In  Figures  8 to  11, 
providing  summary  data  at  ship  and  exercise  levels.  This  process  would  be 
laborious  and  would  probably  require  additional  personnel  to  collect  and  reduce 
the  data. 
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Despite  the  limitations  of  a partially  automated  DCS,  the  evaluation  of 
candidate  performance  measures  clearly  demonstrates  that  such  a system  is 
feasible  and  would  be  beneficial. 

An  Automated  DCS 


If  the  data  obtained  in  a partially  automated  DCS  demonstrate  the  overall 
benefits  expected,  a more  completely  automated  system  might  be  considered. 

The  more  automated  system  would  include  the  following  capabilities: 

1.  Computer  programming  to  automatically  monitor  and  record  all  console 
control  settings,  evaluate  all  settings,  and  record  all  procedural  errors. 

2.  Development  of  automated  voice-recognition  systems  to  monitor,  evaluate, 
and  record  all  communications  errors  for  STG  personnel. 

Methods  for  automatically  recording  procedural  errors  could  be  developed 
along  with  methods  for  collecting  accuracy  and  time  data.  However,  this  would 
entail  more  extensive  programming  to  monitor  control  settings,  and  the  develop- 
ment of  subroutines  to  evaluate  the  control  settings.  It  would  also  require 
more  extensive  modifications  to  the  14A2  Trainer  Complex.  The  software  logic 
necessary  to  identify  control  positions  as  errors  and  to  evaluate  the  significance 
of  the  errors  must  still  be  developed.  An  analysis  of  current  instructor  practices 
in  evaluating  procedural  errors,  appropriate  operational  systems  doctrine,  and 
functional  design  would  provide  the  baseline  for  this  development. 

In  the  following  discussion  of  methods  for  implementing  a DCS,  consideration 
will  be  given  to  the  development  of  techniques  for  automated  recording  of  pro- 
cedural errors.  However,  the  development  of  techniques  for  the  recording  of 
communication  errors  by  automated  voice-recognition  represents  a technological 
gap  in  current  state-of-the-art  capabilities  that  is  not  likely  to  be  filled 
before  the  initial  implementation  of  a PPAS;  consequently,  such  techniques  are 
not  discussed  in  this  report.  For  a discussion  of  potential  for  automatically 
recording  communications  errors,  see  Bell  (1978). 

Methods  of  Implementation 

A fundamental  requirement  for  automated  performance  measurement  in  the  14A2 
ASW  Team  Trainer  Complex  is  that  data  must  be  accessible  not  only  from  the  14A2 
Trainer,  but  from  the  sonar  trainer  that  is  in  joint  mode.  This  is  necessary 
for  both  sonar  and  UB  plot  subteam  measurement  data.  Further,  assessment  of 
ASAC  (Anti-Submarine  Air  Controller)  and  CIC  data  is  required  to  interpret  sonar/ 

UB  plot  data.  Various  options  for  automated  performance  measurement  are  feasible 
This  section  will  address  those  options  from  the  viewpoint  of  providing  a DCS 
fully  integrated  into  the  14A2  complex.  The  current  14A2  complex  configuration 
provides  the  basic  architectural  structure  governing  the  approaches  to  a 
performance  measurement  data  collection  system,  and  dictates  the  hardware 
and  software  requirements  for  implementation  of  various  automated  performance 
measurement  options. 
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A block  diagram  of  the  14A2E  complex  at  FLEASWTRACENPAC  is  shown  in 
Figure  12.  Four  sonar  trainers  are  currently  integrated  into  the  complex: 

14E19,  14E23,  14E24,  and  14E27.*4  Each  of  these  trainers  interfaces 
with  the  14A2  Tactical  Trainer  through  the  Universal  Interface  Unit  (UIU). 
Transfer  of  information  from  the  14A2  to  the  trainers  is  via  the  14A2E's 
distributor  output  table.  The  14A2  transfers  a 110-word  message  to  the 
UIU.  The  message  contains  environmental  data,  vehicle  position  and  motion 
parameters,  weapon  parameters,  and  fire  control  status  data.  The  UIU 
transfers  a 140-word  message  to  all  sonar  trainers.  This  message  consists 
of  the  original  110-word  transfer  from  the  14A2,  plus  cursor  range  and 
bearing  position  from  the  sonar  trainer  selected  for  joint  operation. 

The  transfer  occurs  at  a 1-second  rate,  and  only  when  a joint  mode  con- 
figuration exists. 

Transfer  of  information  from  the  other  trainers  to  the  14A2  occurs  in  a 
similar  manner.  For  example,  range  and  bearing  data  from  the  14E19  are  stored 
in  a 96-word  distributor  output  table  before  being  transferred  via  a direct 
multiplexer  control  (DMC)  channel  to  the  UIU,  where  A-scan  and  B-scan  range  and 
bearing  are  extracted  and  the  data  are  converted  from  digital  to  synchro  informa- 
tion to  provide  fire  control  and  cursor  range  and  bearing  inputs  for  joint  mode 
message  reconstruction.  Data  are  transferred  from  the  14E23,  14E24,  and  14E27 
Trainers  in  the  same  manner  but  with  varying  message  word  lengths  and  repetition 
rates. 

For  clarity,  the  discussion  that  follows  will  focus  on  the  14E19/14A2  Trainer 
configuration.  Modifications  to  the  14E23,  14E24  and  other  sonar  trainers  would 
be  analogous  to  those  required  for  the  14E19  Trainer. 

The  14A2  Trainer  displays  all  vehicle  position  and  motion  data,  sonar 
(SQS-23)  and  fire  control  switch  positions,  and  radar  and  seascope  CRTs  at  the 
instructors'  consoles  in  the  problem  control  room.  The  only  error  data  directly 
displayed  are  those  relating  to  time  for  attack.  These  displays  are  rarely  used. 
The  14A2  Magnetic  Tape  Unit  (MTU)  is  capable  of  recording  all  vehicle  position 
data;  however,  no  error  data  are  recorded  or  stored  by  the  computer. 

The  14E19  Trainer  displays  all  SQS-26  sonar  console  switch  positions  and 
range,  bearing,  and  doppler  error  at  the  14E19  instructor's  console;but  its 
computer,  like  that  of  14A2  Trainer,  does  not  record  error  data.  Nevertheless, 
all  14E19  Sonar  data  required  for  performance  measurement  including  range,  bear- 
ing, and  doppler  error,  are  readily  accessible  from  the  96-word  buffer  from  which 
data  are  transferred  to  the  UIU.  Similarly,  most  14A2  data  can  be  obtained  from 
the  UIU  140-word  transfer  buffer. 

The  following  sections  discuss  hardware  modifications  to  the  14A2  Complex 
that  would  be  required  to  implement  various  FMS  capabilities. 

Hardware  Modifications 


The  hardware  modifications  that  would  be  required  to  implement  automated 
performance  measurement  will  be  addressed  at  two  levels.  The  first  level  includes 
mandatory  changes,  or  changes  that  would  be  required  irrespective  of  the  options 
available  in  implementing  automated  performance  measurement. 

^The  complex  also  contains  a 14A2A  Trainer  that  is  similar  to  the  14A2E 
except  that  the  14A2E  has  some  additional  instructor  displays  and  a modified 
Main  Trainer  Program. 
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Since  the  entire  96-word  message  from  the  14E19's  distributor  output  table 
is  transferred  to  the  UIU,  it  is  probable  that  no  further  modifications  are 
necessary  in  the  14E19  Trainer;  however,  modifications  to  the  UIU  would  be 
necessary.  UIU  logic  circuitry  must  be  expanded  to  extract  additional  sonar 
data  from  the  96-word  transfer  message,  including  sonar  range,  bearing,  and 
doppler  error,  and  all  console  switch  positions.  A second  modification  would 
be  necessary  to  permit  the  UIU  to  communicate  with  the  XDS-930  computer  via 
either  the  14A2  general  interface,  or  to  provide  a separate  computer  dedicated 
to  automated  performance  measurement  functions.  All  other  modifications  are 
optional. 

A fundamental  question  is  where  a DCS  program  should  reside.  The  current 
14A2  Main  Trainer  Program  could  be  expanded  to  incorporate  a parallel,  integrated 
DCS  program.  The  main  advantage  of  this  approach  is  that  it  would  save  the  cost 
of  a second  computer  and  make  data  already  resident  in  the  14A2E  core  readily 
accessible.  However,  there  are  several  disadvantages; 

1.  The  14A2  Main  Trainer  Program  would  be  extensively  affected,  and  a 
separate  DCS  program  would  be  required  for  the  14A2A  trainer  (and  for  each  14A2 
model  at  other  sites  implementing  a DCS,  since  the  main  trainer  programs  differ). 

2.  The  XDS-930  Computer  has  only  16K  words  of  memory.  This  is  probably 
insufficient,  and  expanded  memory  core  size  would  most  likely  be  required  for 
continuous  data  storage  and  to  drive  optional  peripherals. 

3.  This  approach  precludes  the  capability  of  growth  in  which  performance 
measurement  data  could  be  gathered  from  two  trainers  running  in  independent 
mode.  (This  capability  is  desirable  during  some  advanced  training  exercises.) 

A more  likely  approach  to  implementing  a DCS  is  shown  in  Figure  13.  A 
separate  computer  dedicated  to  automated  performance  measurement  would  allow 
considerable  flexibility  in  both  system  design  and  growth  potential.  The  fol- 
lowing advantages  would  be  gained: 

1.  Modification  to  current  system  hardware  would  be  limited  almost  exclusively 
to  the  UIU.  This  would  limit  the  effect  on  the  ongoing  utilization  of  the  14A2 
including  documentation,  maintenance,  and  operational  procedures. 

2.  The  14A2  Trainer  could  be  updated,  or  replaced,  and  the  new  modification 
or  new  equipment  could  be  more  easily  integrated  with  the  established  DCS. 

3.  Partially  automated  performance  measurement  data  collection  could  be 
implemented,  with  provisions  for  continued  expansion,  and  such  expansion  would 
be  independent  of  14A2E  operations  currently  being  conducted. 

4.  The  DCS  system  could  be  easily  expanded  to  permit  data  collection  from 
both  the  14A2E  and  the  14A2A  Trainers. 

In  order  to  precisely  determine  the  most  cost-effective  approach  to 
implementing  the  type  of  system  shown  in  Figure  13,  it  would  be  necessary 
to  conduct  studies  that  would  assess  the  computer  processing  capabilities 
and  memory  size  necessary  to  store  the  software  required  to  compile  all 
appropriate  performance  measurement  data  and  to  provide  input/output  control 
of  the  peripherals  that  would  be  incorporated  into  the  DCS. 


47 


L 


TRAINER 

(SQS-35) 


A corollary  issue  is  the  possible  use  of  a disk  device  to  store  data  in- 
stead of  using  computer  memory.  In  order  to  collect  continuous  data  such  as 
sonar  and  TMA  error  data,  it  is  probable  that  a disk  would  be  necessary.  The 
disk  is  a mass  storage  device  that  can  be  used  in  either  of  two  ways.  First, 
it  can  be  used  to  store  raw  data  that  could  later  be  accessed  by  the  CPU  for 
appropriate  reduction,  formatting,  and  output  to  peripherals.  Second,  it  can 
be  used  to  store  already  formatted  data  for  subsequent  output.  The  method 
adopted  would  probably  depend  upon  how  rapidly  data  are  received  from  the 
interface. 

The  only  required  peripheral  devices,  other  than  a disk  drive,  are  (1) 
an  automatic  send /receive  (ASR)  teletype  terminal  or  an  alphanumeric  keyboard/ 
CRT,  and  (2)  a high  speed  printer  (HSP).  Optional  peripherals  include  (1)  a 
magnetic  tape  unit  (MTU),  and  (2)  an  X/Y  plotter.  Each  of  these  peripherals 
provides  unique  advantages  for  data  input/output  and  enhanced  performance 
measurement  capability. 

The  ASR  terminal  would  serve  as  the  basic  input /output  device.  An  ASR 
terminal  can  be  used  to  print  summary  statistics  data;  however,  this  would 
provide  slow,  relatively  inefficient,  and  limited  printout  capability.  If  a 
keyboard/CRT  terminal  were  to  be  used,  the  data  would  have  to  be  printed  by 
HSP.  The  terminal  would  be  used  for  several  purposes: 

1.  To  enter  personnel  social  security  numbers  and  assigned  billet  stations 
for  each  exercise. 

2.  To  enter  a system  configuration  mode  instruction  word.  This  instruction 
word  would  tell  the  DCS  computer  which  trainers  are  being  used  (for  example, 

14A2A  independent,  or  14A2E  and  14E19  in  joint  mode)  and  thereby  define  for  the 
system  the  data  that  are  to  be  accessed  and  stored. 

3.  To  control  other  peripherals,  such  as  the  HSP,  by  specifying  the  desired 
data  to  be  printed  out. 

4.  To  run  DCS  utility  programs  for  maintenance  of  the  system. 

The  HSP  is  required  for  efficient  data  printout.  Although  data  could  be 
printed  on  an  ASR  terminal,  the  jISP  would  allow  much  faster  printout  and  pro- 
vide greater  flexibility  in  format.  Since  PPAS  data  will  probably  have  to  be 
recorded  on  tape  and  be  integrated  into  a central  data  base,  an  HSP  could  be 
provided  for  the  central  data  base  computer.  This  would  make  it  unnecessary 
to  provide  an  HSP  at  each  training  center. 

PPAS  data  would  be  stored  permanently  on  tape  through  the  magnetic  tape 
unit  (MTU).  It  might  be  possible  to  use  one  of  the  trainer's  MTUs  to  store 
data,  as  is  shown  in  Figure  13.  This  could  be  accomplished  only  if  the  MTU 
is  compatible  with  the  PPAS  computer  and  with  the  MTU  of  the  central  PPAS  data 
base.  Otherwise,  each  training  center  would  be  required  to  have  an  MTU  dedicated 
to  the  PPAS. 


The  X/Y  plotter  would  provide  a means  for  printing  out  graphs  of  error  data. 


Software  Modifications 


The  temporary  data  collection  programs  and  data  reduction  programs 
developed  for  this  research  demonstrate  the  feasibility  of  developing  soft- 
ware for  a permanent  performance  measurement  data  collection  system.  Currently, 
there  is  no  programming  provided  in  the  main  trainer  programs  of  the  14A2E 
(or  14A2A)  to  collect,  store,  reduce,  or  print  performance  measurement  data. 

The  limitations  of  the  temporary  data  collection  programs  can  be  overcome,  and 
programs  could  be  developed  to  obtain  valuable  performance  measurement  data. 

If  an  approach  to  integrating  a PPAS  DCS  similar  to  that  illustrated  in 
Figure  13  is  taken,  a comprehensive  computer  program  must  be  developed.  The 
overall  program  structure  would  be  dependent  on  the  design  of  hardware  modifica- 
tions, and  specifics  cannot  be  delineated  at  this  time.  However,  an  analysis 
of  the  various  performance  measures  to  be  obtained  and  the  uses  to  which  they 
will  be  put  dictates  the  following  functional  requirements  for  the  computer 
program: 

1.  Monitor  and  record  all  control  positions  for  the  appropriate  sonar 
and  UB  plot  consoles,  and  record  times  at  which  changes  occur. 

2.  Provide  subroutines  to  evaluate  control  position  status  and  changes 
based  on  current  tactical  conditions  or  related  system  configurations. 

3.  Record  range,  bearing,  and  doppler  error  at  each  NDT  time,  and  store 
these  data  for  reduction  and  scoring  after  the  exercise  is  completed. 

4.  Record  TMA  course  and  speed  error  during  the  appropriate  time  intervals. 

5.  Store  TMA  error  data  for  reduction  and  scoring  after  the  exercise  is 
completed. 

6.  Monitor  and  record  total  contact  time. 

7.  Record  all  fire  control  status  data. 

8.  Provide  a general  executive  routine  controlling  all  data  access,  transfer, 
and  processing. 

9.  Control  and  drive  data  storage  and  retrieval  on  a disk  device. 

10.  Control  peripherals  for  printouts  of  all  required  performance  measurement 
data. 


CONCLUSIONS 

1.  Data  from  the  1AA2  ASW  Team  Trainer  Complex  can  provide  useful  inputs 
to  a Performance  Proficiency  Assessment  System. 

2.  Procedures  can  be  developed  for  summarizing  and  presenting  proficiency/ 
del iciency  data  so  that  these  data  can  be  readily  understood  by  Navy  personnel 
managers. 

3.  Procedures  can  be  developed  for  automating  the  collection  and  analysis 
of  the  desired  information. 
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FUTURE  ACTIONS 


On  the  basis  of  the  information  provided  in  this  report,  NAVPERSRANDCEN 
will  take  these  actions: 

1,  Experiments  will  be  carried  out  to  determine  the  degree  to  which  the 
types  of  performance  information  described  in  this  report  would  be  useful  as 
part  of  a Performance  Proficiency  Assessment  System.  The  performance  data 
recommended  in  this  report  will  be  gathered  from  a sample  of  approximately 

100  ASW  teams,  summarized  in  a manner  similar  to  that  shown  in  Figures  3 to  11, 
and  presented,  along  with  any  additional  data  that  may  be  required,  to  appropriate 
decision  makers.  For  these  experiments,  the  desired  data  will  be  gathered  using 
refined  versions  of  the  procedures  followed  to  gather  data  for  this  report. 

2.  If  it  is  determined  that  such  information  is  useful,  the  steps  will  be 
identified  that  must  be  taken  to  automate  the  collection  and  analysis  of 
performance  data  from  the  14A2  ASW  Team  Trainer  Complex.  The  automation 
would  be  based  upon  the  procedures  and  techniques  outlined  in  this  report. 
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